DOCONENT RESUME 



ED 310 737 



IR 013 930 



AUTHOR 
TITLE 



Moonerir Jef; And Others 

The Evaluation of a National Research Plan To Support 
the Implemertation of Computers in Education in the 
Netherlands. 
Kar 89 

19p,; Paper presented at the Annual Meeting of the 
International Association for Computers in Education 
(San Francisco, CA, March 27-29, 1989) • 
Speeches/Conference Papers (150) — Reports - 
Evaluative/Feasibility (142) 



PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage, 

*Computer Assisted Instruction; *Educational 
Innovation; *Educational Planning; *Evaluation 
Problems; Feasibility Studies; Foreign Countries; 
Information Technology; Inservice Teacher Education; 
*National Programs; *Research Proposals 
*Netherlands 



ABSTRACT 



This paper describes the evolution of a national 



research plan for computers and education, an approach vhich was 
initiated in the Netherlands in 1983 • Two phases can be recognized in 
the Dutch experience: one from 1984 until 1988 and one from 1989 
until 1992. Building upon the experiences of the first phase, 
research plans for the second phase are outlined. General conclusions 
are that research plans should be formulated within a concrete and 
well-defined framework, oriented more toward implementation issues 
than toward theoretical research qiestions and undertaken in a 
context of international cooperation. Priorities for the next phase 
of research include: (1) procedures and criteria for software 
evaluation by teachers; (2) alternative approaches to v.he inservice 
training of teachers in computer usage; (3) computer applications in 
vocational education; (4) cooperative learning and computers; (5) the 
effect of computers on higher cognitive skill development; (6) the 
applicability of artificial intelligence techniques to the production 
of courseware; and (7) the development of interactive coaches. 
(Autnor/GL) 



* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



The Evaluation of a National Research Plan to Support the 

Implementation 
of Computers in Education in the Netherlands 



Jef Moonen , Betty Collis , Klaas Koster* 



us OEPAm-MKNTOF KDUCATION 

Oflice o* Education*! ResetrcM tnd U-nprOverr^ent 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

V/ This document his been reproduced as 
received from the person or orgemzition 
onginiting \\ 
D Minor Changes hive been mide to improve 
reproduction gullity 



Points 0* vie* or opinions StileO in thiS docu 
ment do not necessarily represent ofliciai 
OERI posilion or policy 



* Department of Education, University of Twente, PO Box 217, 7500 AE 
Enschede, The Netherlands 

Department of Education, University of Groningen, Westerhaven 16, 9718 
AW, Groningen, The Netherlands 



Paper presented at the lACE '89 Conference 
San Francisco, Maich 27-29. 1989 



■PERK'ISSION TC rtEPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

Bettv A. Collis_ 



bt5l U)KY AVAlLAbLi: 



TO THE EDUCATIONAL RESOUBCES 
INFORMATION CENTER (ERIC)/' 



Abstract 



This paper describes the evolution of a National Research Plan for 
Conputers and Education in the Netne>-lands . This approach was initiated in 
1983. There are two phases that can be recognized in the Dutch experience: 
one from 1984 until 1988 and one from 1989 until 1992. The paper described 
the research plans for the second phase, based upon the experiences of the 
first, and draws some general conclusions. These conclusions are: 

- Research plans should be formulated within a concrete and well defined 
framework. 

- The present orientation of research plans in this area should focus much 
more upon implementation issues than on theoretical research questions. 

- There is ? need for international cooperation in this area. 



1 . BaCMround 



1.1 Organisatio n of Educational Research in the Netherlands 

Educational research in the Netherlands is executed by universities 
after being initiated either by them or the Institute for Educational 
Research (SVO) . This Institute does not perform the research itself but 
invites universities to apply for research projects. A formal evaluation 
and decision procedure is followed after applications are sent in to the 
SVO. In order to create research frameworks for different aspects of 
education, SVO regularly gives assignments for the execution of so-called 
'problem exploration studies'. Such studies are focussed toward a certain 
educational area and result in inventories of crucial research approaches 
and research questions. 



1.2 Introduction of Computers in Dutch Education 

In 1984 the Dutch government launched a five-year Informat'.cs 
Stimulation Plan, the INSP. The major emphasis of the plan was to create a 
momentum for the introduction of computers in business, research and 
education. A specific part of the plan focussed on stimulation of the area 
of computers and education and had a budget of approximately 160 million US 
S. Projects were set up to support the introduction of computers in 
different educational sectors: primary, secondary (general and vocational), 
and special education. Special programs were initiated to start the 
development of a national and regional infrastructure and methodology for 
courseware development. Much errphasis was put on teacher training 
(inservice and preservice) . A very small part (2.5%) of the plan was 
related to research. 

The INSP ended in December 1988, and has been followed by another 
five-year plan: the PRINT-pro ject . The major emphasis of this plan is to 
continue the efforts started during the INSP, and to specifically focus on 
the implementation of computers into education- The budget for this plan is 
even higher than the INSP-budget (250 million US $) . 

2. Initial Research on Education and Computers 



2,1 The First Problem Exploration Study 

Although research on computers in education started in the 
Netherlands at the end of the 1960s, the first systematic problem 
exploration study in this area was carried out in 1984. The report was 
discussed in a broader context at a working conference at the end of 1984. 
The intention of that conference was to formulate the foundation of a four- 
year plan for research to be executed during the period 1985-1988. In 
reality, however, much discussion carried on after the conference about the 
choices that had been made for specific research lines. The great variety 
of suggested research topics and limitations in the budget added to the 
general confusion. As a result a great diversity of research projects was 
carried out during the period 1985-1988. The lesson to be learned was that 
a research plan should be developed in a clearly stated and specific 
context, in order to avoid 'fundamental' discussions which can result in 
confusion and endless compromises. 



2.2 Rese arch during the Period 1985-1 Qflfl 



In this period 42 research projects were executed. These projects 
were originated from three directions: the official program line of SVO 
about computers and education (18), the research plan connected with rhP 
Informatics Stimulation Plan (24), and research projects suggested by 
individual researchers (3) . 

An analysis of these research efforts show little coherence among 
them. However three clusters of activities emerge: 

- Feasibility studies, surveys and evaluation research (20 projects). 

- Research and development of educational software (8) . 

- Theoretical research (14). 

2.2.1 Feasibility Studies and Surveys 

At the start of the period 1985-88 there was great need for 
feasibility studies and surveys, to get a better idea about the extent of 
initial exploratory wor)c. A great emn-hasis was also placed on the 
preparation of the so-called 'Experimental Schools' Project. This is a 
project in which research groups at universities wor)c closely with one or 
two secondary schools to create a kind of practical laboratory in which the 
implementation of computers in the current curriculum can better be 
stuaiod. In the Netherlands, three of these experimental schools have been 
created, two in connection with the University of Twente and a third in 
connection with the University of Amsterdam and the University of utrech^ . 

An examination of the feasibility studies and surveys suggests that: 

- The introduction of the use of computers in schools takes much more 
time than was expected. ,.n 'incubation' time of two ye?rs is felt 
to be minimum. 

- One does not get the impression that surveys have a crucial 
influence on the formulation of governmental policy, when there is 
no direct connection between surveys and follow-up studies, 
especially policy-oriented activities, one car, question the value 
of tiiesQ surveys, especially as they loose their topicality very 
rapidly. 

- Although there was considerable experience in higher education 
about how to use computers in education, the activities of the INSP 
were not related to the research results and experience of this 
sector. This lac)c of cooperation was caused by the fact that 
university education was excluded from the INSP. 

2.2.2 Software Development Research 

A distinction can be made between software development projects whose 
sole goals are to develop products to be used in classrooms and projects 
intended to produce products to support more theoretical research. 
Development research of the second type started much later than the 
feasibility studies and surveys in the INSP and less money was available. 
On the other hand, much money was spent on the first type of software 
development wor)c in school projects. Unfortunately, no connection was 
arranged between these school-oriented 'production' projects and the more 
theoretical 'development- /research' projects. 

When the research-oriented software was compared with traditional 
learning situations in classrooms, its effects on learning outcomes yielded 
no clear results. 



In general the following conclusions can be fomulated about the 
1985-1988 software development activity 

- The complexity of writing computer-aided learning programs beyond 
the drill-and-practice mode is highly underestimated. 

- There was no integration of 'production' projects and 
'development /research' projects. However, each should complement 
the other. 

- There was no adequate softwa^ development infrastructure in the 
Netherlands; too many research institutes want to be involved in 
this area but they lack the hardwcire, the software and professional 
experience to work in an efficient way. Creating a limited number 
of high quality software development units seems to be desirable. 

2.2.3 Theoretical Research 

Theoretical research activities were generally originated by 
individual researchers. One central theme can be recognised in those 
projects: 

The development of effective feedback procedures during the 
interaction of students with a computer program, including the 
construction of intelligent tutoring systems and computer coaches. 

However, no significant progress can be reported. 

Because of the complexity of tnis area, it is highly desirable that 
cooperation among researchers ir the application of artificial intelligence 
techniques to comouter-based ed ation be organised, certainly in the 
Netherlands, and maybe even on an international scale. 



3. Second Problem Exploration Study 

As already stated in section 1.2, the INSP finished at the end of 
1988 and SVO wanted to set up a new research line for education and 
computers for the period 1989-1992. A second problem exploration study was 
therefore announced in early 1988. Based upon the experiences of the first 
problem exploration study, the new research agenda was called to be written 
in a much more specific form. The letter of assignmert of the second 
problem exploration study described this framework as follows: 

'After considerable efforts done in this area by the 
government, especially in relation to hardware 
provisions, teacher training and courseware 
development, there is a general feeling that ^he time 
has come to get better and more precise insight in the 
effects of the invested money. Therefore, a research 
agenda has to be established whose results 
give concrete indications about the effects 
of computers in education, and through which it will 
become possible lo evaluate tne investments of the past' . 

Given this assignment, a new assessment was made of the current 
situation regarding computers and education in schools and a general 
framework was developed into which a new research agenda could fit. Thiy 
new assessment and framework are described in the following sections. 



4, Current Situation in Schools -1988 



As noted in section 3, preparation for the second problem exploration 
study on computers and education began at the end of 1987. To support this 
study, a survey was conducted during 1988 concerning the actual state of 
affairs of the use of computers in Dutch education. The major conclusions 
of the survey were: 

- Although the average number of computers in Dutch secondary schools 
is 23, schools still express a need for more hardware. 

- The current standard in Dutch secondary education is MS-DOS. 
Schools however prefer a standard that allows more possibilities. 

- The cur'.ent policy is concentrated on the introduction of computers 
at the micro level (referring to direct student interactions) . More 
emphasis should be put on the meso- e.id macrolevel (relating to 
supporting the teacher and to school organisation) and the mutual 
relations between these levels. 

- Interest for computer-aided learning is not the same in the 
different school sectors. In the vocational sector there is little 
interest in computer-aiaed learning . 

- The choice of courseware by schools is not very well structured. 

- Development of courseware by teachers is seen less frequently. 

- Besides the need for a computer laboratory at ^chool, there is a 
growing need to also have computers available outside this 
laboratory. 

- Schools need a systems person to take care of the daily maintenance 
of uhe computer systems. 

- The potential influence of computers on school organisation and 
school management has to be better realized. 

- Inservice training of teachers should focus much more on subject- 
oriented training; it is necessary to develop creative solutions to 
solve the problem of the huge numbers of teachers to be trained. 

- Teachers feel they have insufficient information dissemination 
about computers in education. The transfer process regarding such 
information needs a much more explicit approach. 

Of these points, the major complaints expressed by teachers are about 
lack of support and lack of information dissemination and transfer. In 
addition they wish to be better able to evaluate courseware themselves and 
to get concrete suggestions about how to integrate computer usage in the 
curriculum. 



5. A General Framework 

Based on these survey results (section 4/ . previous research from 
1985-1988, and more general experience, a conceptual framework was 
developed to guide the further work of the second problem exploration 
study. It was first decided that this conceptual framework will not be one 
which focuses directly on student perfonrance. This may seem surprising, 
given the prime irt^ortance of learner outcomes in any consideration of the 
impact of computers in education. However, there are two major sets of 
reasons that make systematic assessment of learner outcomes, presumably as 
caused or mediated by a computer application within the learning 
experience, limited in effectiveness at this time. Vhe remainder of this 
section summarizes the rationale for the conceptual framework and gives a 
brief description of its general components. 
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5,1 Rea30n3 for Difficulty in Assessment of Impact of Compute rs on Learning 



There are at least two sets of reasons which make direct assessment 
of the impact of a specific computer intervention difficult at the present 
time. These relate to the complexity of the computer-use environment and to 
measurement aif f iculties . 

5.1.1 Complexity of Computer-Use Environment 

One of these sets of reasons relates to the rapidly changing nature, 
and relative immaturity (with relation to the support of computer-learning 
experiences), of the environment surrounding computers in schools. For 
example, teachers vary substantially in their knowledge and expertise with 
regard to either computer management in itself or its pedagogical 
application and integration. Learning resources to be used in conjunction 
with new technologies also vary substantially in many ways, such as in 
characteristics of the user f nvironments; these variations will have a 
strong effect on the subsequent deployment and impact of computer-related 
experiences on the learner. Characteristics of the surrounding learning 
environment itself, ranging from social and physical characteristics ot the 
situation as experienced by the individual learner to the support offered 
his teacher from either the immediate schoool environment or provided by 
the larger external educational support system, also have a substantial 
influence on eventual learner impact. 

These three major sets of influences ~ teacher, materials, and 
surrounding organization — are highly multifaceted and together form the 
boundary parameters within which all learning impact associated with 
computer use occurs. A major reason why so much current research activity 
attempting to illuminate learner outcomes as mediated by computers is 
inconsistent or unconvincing is that the large;, network which saturates and 
shapes this mediation is treated as if it were a homogeneous constant. 
Before we can fairly comment on the impact of computers on the learner we 
must know more about the influence of this surrounding network, at least to 
the extent of bsing better able to systematically elaborate its most 
salient characteristics relative to the subsequent impact of computer 
experiences . 

5.1.2 Limitations in Measurement Strategies 

A second reason for postponing a direct focus on learner outcomes for 
the current phase of SVO-supported research activity is the problem of 
measurement. Reliable and valid assessment criteria, and i.istrumentation to 
measure the extent of change relative to those criteria, are not well 
established. Questions concerning the relative advantage of one approach to 
teacher training compared to other approaches, for example, cannot be 
answered in an efficient or valid way without prior agreement as to what 
standards are to be used for "success" and how they are to be 
systematically and reJiably applied. Criteria relating to cost-benefit 
analyses are particularly useful, yet no common procedure exists to 
determine a basal framework for cost-benefit quantification, without 
improved strategies for assessment of both process and outcomes — not only 
of student performance but also of auxiliary variables such as those 
relating to materials development, teacher support, and organizational 
variations — assessment will be limited to imprecise and therefore 
unproductive appraisals with little generalizability . Thus the second 
overiding concern of the conceptual framework for the second problem 
exploration study is a focus on appropriate, manageable strategies for 
sensitive and valid measurement of variation with respect to critical 
points in the computer-impact system. An intention to focus directly and 
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iznmeciiately on learner outcome variables without the tools to sharpen or 
even decode what is seen through the lens of that focus may at the least be 
premature and at the more serious lead to faulty conclusions. 

Based on this rationale (a rationale which is implicitly confirmed by 
the results of the field surveys in section 4) . Figure 1 shows the 
simplified conceptual organization that was suggested for the 1989-1992 
phase of computer-related research activity: 



How to measure effectiveness? 
cost-benefit effectiveness? 




Teacher Training 
Support 





Organization 

of the Learn 
Environment 




Issues Associated with 
Acquisition/Development 
of Materials 



Figure 1. A general conceptual framework for research activity on the 
impact of con^uters in the school setting. 



5.2 Elaborated Research Agendas: An Overview 



Given this conceptual framework diagrammed in Figure 1, research 
agendas can be organized in a variety of ways. Based on the survey 
responses obtained for this study and summarized in section 4, six general 
areas of need emerged. These field-based needs can be summarized as: 
software suitability assessment, teacher training, teacher support, school 
and instructional organization, measurement and evaluation issues, and 
software development and acquisition. Each of these areas was examined in 
some detail and suggested research topics discussed. The following sections 
summarize the discussions that appeared in the second exploration study. 

5.3 Software Assessment 

Teachers consistently indicate that they feel they have inadequate 
resources for effective evaluation of software. From the teachers' 
perspective, their major needs are resource-related: they indicate they 
need more time for software evaluation, more software to evaluate, more 
access to hardware (both at school and at home) for software evaluation 
purposes, and more technical support to help them manage unfamiliar 
software. A related issue is who should organize and support these 
resources — the school? the region? the educational support system? 
perhaps even the software or hardware vendors? 

Research, however, indicates a more fundamental problem in connection 
with software assessment. This problem relates to the criteria around which 
such assessments are based. There is no guarantee that, given time and 
resources, teachers will make insightful educational appraisals of the 
software they are inspecting. On the contrary, various studies show that 



9 



teachers vary considerably in their perspectives with regard to software 
suitability, and f ur»dam9ntal to this variation is a lack of cons'insus on 
appropriate criteria for evaluation. These criteria can relate to both 
technical and pedagogical considerations but also to management and cost- 
effectiveness perspectives. Teachers benefit from guidance and training in 
the use of a variety of evaluation criteria and in instruments that 
encapsulate those criteria. The task is made more difficult by the need for 
different perspectives for different types of software — drills, 
tutorials, simulations, data capturing and analysis, open-ended tool 
products, "real-world" products such as applications packages, and 
specialized software for vocational education. Strategies for priorizing 
evaluation criteria are also important — what characteristics are most 
critial in terms of suitability appraisal? The research literature about 
software evaluation largely reflects work dene in the early 198 0s before 
the emergence of many of the types of open-ended or tool-type software now 
considered so important in educational settings. New work is necessary on 
efficient and effective methods for appraising the contemporary variety of 
educationally applicable software as well as on techniques for training 
teachers to apply these methods, given the typical constraints on teachers' 
time and on resources available for evaluation. 

Other issues are also important. Who should provide t^'^ training for 
teachers in software evaluation? Who should train these trainers? what 
influence can be brought to bear on manufacturers for the provision of 
representative demonstration disks that can allow teachers to interact 
adequately with a variety of software before purchasing decisions are made? 
Also, what is the role of external sources of software evaluation, such as 
teachers' journals? How well referred to are printed reviews of software? 
Is there a place for the use of media for creative new approaches to 
software evaluations (such as providing videotapes or even interactive 
videodisks, available at centralized resource centres, that highlight 
features of a variety of software packages dealing with the same general 
topic) ? 

In summary then, the following questions need further research and 
offer both theoretical and practical payoff: 

5.3.1 what are effective criteria for the evaluation of different 
types of software for use in educational settings? The 
criteria should consider: 

- potential instructional impact relative to other 
methodologies 

- technical features, such as user interfaces and options for 
teacher adaptation of program content 

- cost-effectiveness and cost-benefit 

- implementation considerations, such as management challenges 
in the classr'^om setting 

How can these criteria be most efficiently assessed? 

5.3.2 How can teachers be trained to apply these criteria? How can 
we measr.re the effectiveness of the training? Who should 
organize, staff, and finance the training? In what situations 
might it be more effective for teachers to use published 
evaluations of software from external sources than to perform 
their own evaluations? 

5.3.3 In what ways can technology be used to expand the variety of 
methods available to teachers for software appraisal? 
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5.3.4 What types of local organizational support can be most helpful 
to teachers in facilitating their software evaluation 
activities? who can most effectively take the init-i::tlve in 
procuring software for evaluation ar^-^viLies? what sorts of 
arrangements c^n rriade oetween educdtion authorities and 
software distributors to maximize teachers' opportunities for 
convenient access to software for evaluation? Should released 
time for software assessment be made available to teachers? 
How much time is reasonable? 

Cluster 5.3.1 can be considered at the theoretical level; the other 
clusters relate more to applied research. The experimental schools could 
serve as testing sites for the organizational and developmental ideas 
identified in these other clustars. 



5.4 Teachers and Computer^Related Training 

5.4.1 Curre nt Training Perceived as Inadequate 

As was found with respect to the specific issue of software 
evaluation, the results of ^:he survey in section 4 indicate that teachers 
more generally perceive their training with respect to conputers to be 
inadequate and even inappropriate. Teachers both in and outside of The 
Netherlands often feel that their current training in this area fails to 
give them strategies for the effective use and management of computers in 
their classrooms, sometimes because the training is too brief, or is 
irrelevant, or is ineffectively or inconveniently delivered. 

5.4.2 Pertinent Prior Research 

These problems are not new. A considerable amount of discussion has 
been occurring since the early 1980s witn respect to appropriate content 
and orocedures for introdacing teachers to the educational uses of 
computers. Debate continues to flourish around a number of different 
issue."5, particularly issues relating to the cortent of such training and to 
appropriate models for ics delivery. Cost-effectiveness considerations — 
given the large number of teachers requiring training, the different 
experience levels and needs of these teachers, and the continually changing 
nature of the field for which the training is meant to be pertinent ~ have 
not yet identified a recommended model for the teacher training, either 
before or after the teacher is a practitioner. 

However, some consensus has emerged with respect to these issues. Fox 
example, it is now considered appropriate that all teachers have some basic 
formal training with computers, although there is no agreement as to if 
this exposure should occur during preservice training, and if it does, at 
the expense of what other portion of ^he overall teacher training program. 
As to content, a general agreement is emerging that the majority Oi. 
teachers do not want or need instruction in how to program or in how to 
generate their own software with authoring tools. 

5.4,3. Training for Teachers as PAr<^r> nal Users of Technology 
In addition, there is consensus about the value of teachers learning 
word processing and becoming familiar with representatives from each of the 
major areas of educational software such as simulations, drill and 
tutorials. The balance and sequence, however, of teacher familiarization 
with these different aspects of computer-related Graining has not been 
uniformly established. There is some evidence thac initial computer 
experiences eit¥)hasizing the teacher's own use of the computer as a personal 
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tool may prove to be effective in increasing the likelihood that the 
teacher will eventually use a computer for instructional purposes with his 
or her students. 

5.4.4 Alternative Del iverv Strategies for Teacher Traini ng 
Delivery strategies for ongoing teacher training are also being 

critically examned, particularly with regard to organization. Using the 
new media as a vehicle for the delivery of such training experiences offers 
considerable potential, particularly from a cost-effectiveness perspective 
(as, for example, occurs when satellite transmission of specialized teacher 
training courses or seminars allows service to teachers in situations where 
it would be difficult and costly to provide training in their own 
environments) . 

5.4. 5 Suggested Research Questions 

With these considerations in mind, the following focuses are 
suggested as appropriate research topics: 

5.4.5.1 What is the relative effectiveness of different models for 

teacher training in terms of the current needs of teachers in 
The Nethei.lands? 

How well do different models cope with the necessity of 
training large numbers of teachers with different levels of 
skill? How effective are the various models with respect to 
the continually changing nature ut the computer-field? How 
can the impact of different training methods be measured? 

5.4.5i.2 What should the content of teacher training be? 

How much general knowledge about informatics and programming 
should be expected of primary teachers? of general secondary 
teachers? of vocational education teachers? How much of 
teacher training should relate to the development of 
teachers' personal uses of computers for record keeping anc 
word processing? To what extent should teachers be trained uo 
modify software? What is an appropriate balance between 
curriculum applications and informatics in the training 
provided for teachers in the different school sectors? 

5.4.5.3 How can teacher training be most effectively organized? 

Is training more effective on-site or at regional centres? 
Are there alternative methods of training, such as through 
self-help materials or television broadcasts, that can help 
meet the training needs of the large number of teachers in 
The Netherlands needing computer-related training? How will 
the teacher trainers, especi. lly those at teacher training 
institutions, be trained? 



5.5 Oraaniz ^.tional Suppor^ for Teachers 

In addition to training opportunities, a teacher's eventual usage of 
computers for instructional purposes relates to the type of support 
received for such use in his or her work environment. This support may be 
manifested in a variety of ways — through the resources of hardware, 
sofware, time, and space made available to the teacher in order to prepare 
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for con^uter use with students; through the personnel available to the 
teacher '-^r technical as well as strategic support. We have discussed these 
perspectives before in the context of supporting the teacher in software 
evaluation activities. Thsre are, however, a number of issues uhat deserve 
critical examination with respect to I he level of support more generally 
available to the teacher with regard to compter use. 

5.5.1 Perceive d Inadequacies in Support Resources 

It is common, both in and ou' of The Netherlands, for teachers to 
inaicate they would like to do more with computers but that they do not 
have access to adequate resources in order to do so. These resources 
usually include time, hardware, and software, when to find the time to 
adequately prepare lessons incorporating computers or more generally to 
become more confident and competent with one's own level of co*:>outer-using 
facility is a serious problem for many teachers and the reason thoy 
frequently cite as underlying limitations in their eventual use of IT with 
their students. 

Perhaps because it seems to relate more to procedure than to a 
conceptual educational issue, the impact of more or less preparation time 
for teacher deployment of computers for instructional purposes has received 
little focus as a research issue; it is accepted as a truism that more time 
for preparation would result in better preparation and in more effective 
classroom practice. Such an assumption should be looked at more 
systematically, within the context of the investigation of a variety of 
organizational arrangements all aimed at giving t achers more time and 
opportunity to prepare themselves outside of the classroom for comput'ir use 
within the classroom. One frequently mentioned strategy is to make a 
computer identical in model to the machine the teacher has access to at 
school available to the teicher at home for preparation purposes. This 
approach still has limitations — no technical support is available to help 
the teacher if procedural problems arise and the presumption is made that 
teachers will do their computer-related professional development on their 
own time and in competition with other preparation activities in which they 
are already involved. 

5.5.2 issues Re lating to School Technology Support Persons 
Teachers commonly acknowledge that the support they receive from a 

school coxiputer system person is another factor in their eventual usage of 
available technology for instructional purposes. There are a number of 
issues relating to this type of support that need to be investigated. One 
relates to training and upgrading — who trains the support persons? Where 
do they receive sucn training? what sLauld the content of such training be? 
How much of the content should focus on technical updating and how much on 
pedagogical considerations? How can ongoing professioi al support for the 
support persons be most efficiently provided? How can a single support 
person, without being personally overloaded, respond to the practical and 
strategical needs of a staff where a large proportion of teachers wish to 
make fuller use of technology? 

These issues may seem to be exclusively the domain of policy makers 
because of the financial considerations involved in any action regarding 
the issues. However, it is appropriate in a research framework to consider 
the in^lications to the system of not providing adequate support for 
computer systems persons and to advise on the likely consequences of 
different policy strategies regarding these persons as well as associated 
issues related to 'released time' assignment for computer-related support 
activities. 
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5.5.3 Information Transfer 

Moving outside the school setting, the support framework available to 
the teacher through the larger educational support system is also an 
inportant variable in the teacher's eventual use of computers for 
instructional purpose?. Aspects of this larger support system have already 
been commented upon; these include organization for teacher software 
evaluation and for t&acher training more generally. They also relate to the 
need for better information ♦iransfer between teachers and schools ariu 
between teachers and specialists working on innovative applications of 
technology in the classroom. Teachers frequently comment that they feel 
they lack access to appropriate information; however, the problem of 
adequate information dissemination is not limited to teachers, but 
characterizes the situation perceived by individuals throughout the 
professional community involved with the computer in education. How to 
become better aware of and be more systematically informed about pertinent 
developments in a rapidly changing field is a major problem at all levels. 
How to translate these continual changes into appropriate and manageable 
information for practitioners is a major challenge. 

5.5.4 S'jQaested Researach Quebcions 

Given these considerations, various organizational issues are 
recoiTOnended as appropriate focuses of applied research: 

5.5.4.1 What sorts of organizational and professional support are 
most valuable for teachers in terms of their ongoing needs 
for preparation time and professional development in the 
confute- area? 

Can a formula for teacher preparation time be found that 
balances teacher needs with cost realities in the school? How 
can on-site computer systems persons be most effectively 
supported in their work? How can they be adequately trained? 
What are the likely consequences of not having school support 
person^ if resources for them are not available? 

5.5.4.2 How can information transfer between teachers, schools, and 
support agencies concerning computer-related developments and 
icisues be improved? 

What methods of information transfer are most cost-effective? 
How can these types of effectiveness be evaluated? Are there 
special needs relative to computers in the various school 
sectors that are not as well met by the existing educational 
support system as they might be through specialized regional 
or national institutions developed directly to focus on 
teacher support for computer use? How can issues such as 
these relating to infrastructure be most effectively 
investigated? 

5.6 Phy sical and Social Organizational Considerations 
5.6.1 Physical Organization of Hardware 

Research indicates a relationship between level and type of computer 
usage in a school and the physical organization of school hardware (such as 
in computer laboratories or distributed witin classrooms) . Laboratories are 
accepted as necessary and desirable organizational modes for school 
corr^rter resources, yet research shows that the concentration of school 
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resources in a laboratory environment can result in little or no usage of 
the facilities by the majority of teachers in the .ichool . Also, the 
centralized laboratory environment may not be an appropriate locale for the 
integration of computer use as a tool within the context of lessons outside 
the informatics area. The language teacher, for example, who would like his 
students to make intermittent contributions to a data base of vocabulary 
which can later be used by the students foi personalized language drill is 
not serviced by access to a full laboratory away from the classroom 
environment but is better served by having stand-alone hardware immediately 
available in the classroom itself. 

5.6.2 Social Organization of Computer Use 

I'lere is another area of organizational strategy that has shown 
itself to be an important variable in the instructional impact of computer 
use in schools. This area relates to the social organization of student use 
with respect to computers. 

The assumption that an optimal goal is that each student can work 
independently at his or her own terminal has been challenged depending on 
the type of computer-related experience involved. Some types of computer 
interaction are enhanced by group use of a single computer and there is 
some research support for the value of working in pairs rather than alone 
at a cortputer even for drill-type activities. Tbe impact of student 
characteristics as well as instru'^tional objectives on decisions about 
optimal student : machine ratios requires careful research; no simple 
standard of student rmachine ratio can be unilaterally endorsed. Also the 
impact of hardware characteristics themselves on student learning need more 
careful study, especially with respect to innovations such as local area 
networking and access to telecommunications. 

5.6.3 Suggested Research Questions 

Within these perspectives, the following research focuses are 
recommended: 

5.6.3.1 What arc the practical and pedagogical considerations 
associated with different physical groupings of school 
computer resources? 

In what ways does the laboratory configuration of school 
computers impact on student usage and on teacher decisions 
about instructional uses of computers? Does local-area 
networking affect teachers' instructional decisions? 

5.6.3.2 What are the implications of different groupings of students 
with respect to computer use? 

What sorts of learning experience are best presented through 
individual use of a computer? When might pairs of groups of 
students relative to a single computer be more effective 
pedagogically? Can we recommend different student: computer 
ratios for different types of learning situations? How can 
the teacher integrate the use of a single classroom computer 
within the demands of overall classroom management? 
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5.7 Issues Relating to Software Development and Characteristics 



R3search supports the common knowledge that there are many issues 
related to software development or adapatation which require continued 
examination. Some issues are particularly important to vocational education 
teachers who must deal with the choice of using professional-quality 
industrial software or instead of acquiring some kind of simulation or 
educational shell relating to the professional software. 

In addition, there are important areas of theoretical work with 
applications to software development that are potentially too piofitable to 
be overlooked, even in a research orientation which emphasizes 
implementation. These include work with new software environments such as 
hypertext and fundamental research into the development of expert systems 
and more efficient user interfaces. 

5.7.1 Suggested Research Questions 

Major research questions include the following: 

5.7.1.1 In vocational education, what sort of software is most 
appropriate? 

Are students better served by having access to software and 
hardware that duplicates what they will use in the workplace, 
or by using simulations of such materials? 

Can user interfaces be built around professional packages to 
provide tutorial support for students? 

5.7.1.2 What characteristics of the user interface are most critical 
for subsequent student learning? 

To w'lat extent should standardization of user interface 
characteristics occur for the differnt educational sectors? 

5.7.1.3 How can new developments in hypertext, artificial 
intelligance and expert system research be incorporated into 
usable classroom learning tools? 

5.7.1.4. To what extent do teachers wish to adapt software for 
their own situations? 

What sort of software support should be available to 
facilitate this kind of teacher control? 

5.8 Toward the Improved Assessment of the Impact of Computers 

5.8.1 Specific Methodological Problems Associated with the Assessment 
of Student Gains From Computer Use 

The final purpose of all of the initiatives relative to teacher 
training, organizational support^ and software development and distribution 
is to promote more effective student learning than occurs using traditional 
technologies. Yet, there is a continued frustration with the difficulty of 
measuring such gains if or when do they do occur. This is particularly so 
with respect to increases in higher level cognitive activity, such as 
problem solving, critical thinking, and metacognitive processing. These 
dif f iculti^ss assume critical importance when higher level outcomes are 
given as major reasons for extensive expenditures of time and money on 
computer applications in schools. However, existing measurement techniques 
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for student performance relative to computer-based learning experiences are 
often cited as being unr3liable. 

5.8, 2 Methodological Considerations Associated with Overall 
Evaluation of Techno] ogy-Rolated Practice and Policy in Education 
In addition* valid and manageable methodology for the more general 
evaluation of computer-related practice and policy in the school setting is 
as yet little understood or practiced. There are particular problems 
involved with the evaluation of technology-related activity which transcend 
those associated with evaluation of school practice in general. One problem 
relates to the rapidly-changing and exploratory nature of technology 
activity; evaluation should not be postponed until a stable situation 
presents itself for more traditional evaluative considerations, as too many 
important decisions must be made in the intervening short term period. A 
second problem relates to the indistinct nature of objectives with regard 
to technology use in schools — often such use is belief-driven rather than 
outcomes-driven and therefore constructive evaluation apparently lacks a 
focal point. Special consideration should be given to manageable 
methodologies that can give valuable evaluative feedback to teachers and 
decision makers during the current phase of school computer activity. 

5.8,3 Suggested Research Questions 

Because of these assessment limitations, we are frustrated in 
producing effective evidence to support the relative impact of one type or 
" rangement of computer- related learning experience relative to another. 
Before sensitive large-scale examination of the impact of various types of 
computer use on student learning can occur we must do more preliminary work 
on the methodology of measuring and interpreting these sorts of impact. The 
following research areas are recommended: 

5.8.3.1 What is an appropriate methodology for school or regionwide 
evaluation of computer-related policy and practice? 

5.8.3.2 How can the cost-effectiveness of different computer (and 
non-computer) applications be calculated? 

How can cost-effectiveness considerations be used as a basis 
for subsequent planning and budgetting in a school or region? 
What are appropriate criteria for cost-effectiveness 
appraisals? 

5.8.3.3 How can we reliably identify gains in higher order thinking 
and how can w argue that such gains transfer from computer 
experiences in school to a broader application? 

5.9 Summary 

Within this framework, we have categorized a set of suggested 
research issues based on current needs in The Netherlands relative to the 
effective application of IT in education. The ultimate goal of the 
recommendations is to improve student learning; however, we believe that 
considerable preparatory work needs to be done before we can expect to see 
such in^rovements . We belive that disciplined inquiry relative to the 
issues identified in this framework can help the advance of this prepatory 
work. 
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6. The Research Proposal 



6.1 Final Proposal 

6.1.1 Influence? fin fhe Final Proposal 

The final research proposal was based not only upon the framework 
described in section 5, but also on a number of other considerations: 

- The budget constraints. 

- The new plans developed in the PRINT-project , the successor of the 
INSP, indicating that activities continued to be organised by 
school sector, and therefore each school sector (primary, general 
secondary, vocational) should have its share in the research 
proposal. 

- Continuation of the 'Experimental Schools' Project. 

- Stimulation of cooperation in the basic research area, certainly in 
relation to the application of artificial intelligence techniques 
in education. 

- Usefulness of the results to the current educational system. 

6.1.2 Budget for the Research Ac tivity 

The available budget (approximately equal to the research budget 
during the INSP) was split up in two categories: 40% for the Experimental 
Schools Project and 60% for other research projects. As the schools 
participating in the Experimental School Projects also received other 
finances for their computer-related activities, the emphasis on this kind 
of action research is remarkable. It shows the conviction in the 
Netherlands that getting insight into the integration of computers into the 
curriculum will only appear through a global approach in which research is 
directly combined with hands-on experience in real classroom settings. 

6.1.3 Selected Priorities 

Within the budget for regular projects and based on the influences 
described in section 6.1.1, the following priorities for the next phase of 
SVO computer-related educational research were selected: 

- Courseware evaluation by teachers: procedures and criteria. 

- Research toward the effects of alternative approaches to the 
inservice training of teachers; research into the possibilities for 
teachers to use computers at home for preparation of their lessons 
and to make fuller use of student feedback; research toward the 
possibilities of using telecommunications in the context of 
inservice training. 

- Research about the kinds of computer-aided learning felt to be 
useful in vocational education. How can professional packages be 
used and integrated in regular curricula, for instance, using 
embedded training techniques? What kind of computer applications 
should get priority in vocational education? 

- Research about cooperative learning and computers;. 

- Methodological studies on effect evaluation relative to the 
influence of computers on higher cognitive skills and on education 
more generally. 

- Research toward the applicability of artificial intelligence 
techniques to the production of courseware. 

- Development of interactive coaches. 
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6.2 Evolution of the Proposal into a New Research Plan 



After the research proposal was published (Autumn, 1988) , a two-day 
working conference was organised in order to discuss its specifics. At this 
conference representatives of the Dutch educational research institutes 
were present, as well as representatives of the Dutch school inspectorate 
and the schools. 

The conference concentrated its discussion upon two major topics: 

- The value of the F":perimental Schools Project. 

- The value of the implementation orientation of the research 
proposals in contrast to an orientation toward more theoretical 
research issues. 

For the Experimental Schools Project the issues were raised that 
within the available project proposals specific research questions were not 
(yet) clearly formulated, that there was no experimental design available, 
and that there were too many opportunities to spend much time on 
development work (developing courseware, developing inservice training) and 
not enough on research. Although many of these remarks made sense, 
everybody was convinced that a new 'global' and unorthodox approach was 
necessary. Generally speaking, there were some doubts about spending a 
considerable amount of money without a higher probability that practical 
results will appear at the end of the project. The Experimental Schools 
Project was finally supported, but it will have to soon present a more 
concrete research plan and "concrete results." 

The other main discussion point was about the necessity of doing 
applied research in this area versus the need for more theoretical 
research. Some claimed for more emphasis on theoretical research as no one 
can predict what new horizons will be opened by theoretical research 
results, horizons that maybe will change our view about computers and 
education in a radical way. No one wanted to argue againsc this position. 
However, considering the budget constraints and the need for immediate 
results, these interventions did not lead to significant changes in the 
research proposal. 

After plenary sessions, working sessions, and much informal 
discussion, the content and approach of the research proposal as described 
in sections 5 and 6.1 was approved. 



7. Conclusions 

In this paper we have tried to develop the following three points: 

- Research plans should be formulated within a concrete and well 
defined framework. 

- The present orientation of research plans in this area should focus 
much more upon implementation issues then on theoretical research 
questions . 

- There is a need for international cooperation in this area. 

We have used the past and present experiences in the Netherlands to 
support these conclusions. The first phase of the Dutch experience supports 
the first of these points. The second phase is contributing to our 
recognition of the importance of the second point. Both phases of the Dutch 
experience reinforce the complexity of the overall task and support our 
third conclusion. 
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